Comparison of Validity of Classification Made According to DINA Model with Criterion-Referenced and Normative Assessment/Bağıl ve Mutlak Değerlendirme ile DINA Modele Göre Yapılan Sınıflamaların Geçerliğinin Karşılaştırılması
Abstract
The purpose of this research is to examine the validity of classifications by DINA Model as a Cognitive Diagnostic Model and by traditional methods. In order to make comparisons between DINA Model and criterion-referenced and normative models, a measurement tool that belongs to “measurement and assessment in education” class which is appropriate for DINA Model analyses is developed. Properties necessary for answering each item of measurement tool are determined by scholars and the Q matrix which shows item-property relation is prepared considering the compatibility of decisions made by various scholars. Above mentioned measurement tool is applied to 471 undergraduates from the Faculty Education and Arts and Sciences of Ege University. Raw scores of undergraduates are classified according to their success, that is whether they passed the class or failed, through criterion-referenced and normative assessment. And then this classification is compared with other classifications based on DINA Model.
The comparison of assessment by normative assessment and classifications by DINA Model shows that the results are different for 50 undergraduates who failed and 28 who passed the class. As a consequence of the study, it is observed that the inconsistency between the normative assessment and DINA Model is 16.5% for the whole group. In assessments done by criterion-referenced, for the students who failed, two methods give the same results. However, for 87 students who are supposed to pass according to criterion-referenced assessment, DINA Model results show that these undergraduates are not qualified to give the right answers of the items. The incompatibility between assessments according to criterion-referenced and classifications according to DINA Model is calculated as 20%.
References
- Cheng Y.& Chang H.(2007). The Modified Maximum Global Discrimination Index Method for Cognitive Diagnostic Computerized Adaptive Testing. Presented at the CAT and Cognitive Structure Paper Session, June 7
- Cheng, Y. (2010). Improving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method. Educational and Psychological Measurement, 70 (6), 902-913
- de la Torre, J. (2008a). The generalized DINA model framework. Unpublished manuscript. State University of New Jersey.
- de la Torre, J. (2008b). The generalized DINA model. Uluslar arası Psychometric Society toplantısında sunulmuştur, Temmuz, Durham, NH.
- de la Torre, J. (2009a). A cognitive diagnosis model for cognitively-based multiple-choice options. Applied Psychological Measurement, 33, 163–183
- de la Torre, J. (2009b). DINA Model and Parameter Estimation: A Didactic. Journal of Educational and Behavioral Statistics March, Vol. 34, No. 1, ss. 115–130
- de la Torre, J. (2010). The Generalized DINA Model Framework, National Science Foundation Grant tarafından desteklenen araştırma raporu.
- de la Torre, J. , Hong, Y. & Deng, W. (2010). Factors Affecting the Item Parameter Estimation and Classification Accuracy of the DINA Model, Journal of Educational Measurement, Bahar 2010, Sayı 47, No:2, s: 227-249
- de la Torre, J. .& Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika. V69, 3, s. 333-353.
- de la Torre, J. .& Douglas, J. (2008). Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data. Psychometrika. V73, N3, s. 595-624.
- Fagan, T. (2002). Implications of the No Child Left Behind Act for States in the Southwestern Region, Southwest Texas: Educational Development Laboratory, Austin.
- Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
- Haertel, E.H. (1990). Continuous and discrete latent structure models of item response data. Psychometrika 55, 477–494.
- Huebner, A. (2010). An Overview of Recent Developments in Congnitive Diagnostic Computer Adaptive Assessments, Practical Assessment, Research & Evaluation, Vol15, Sayı 3. Ocak
- Huebner, A., Wang, B. & Lee S. (2009). Practical Issues Concerning the Application of the DINA Model to CAT Data, D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.
- Krüger, S. (2003). The PISA-shock and its Consequences: The Future of Libraries for Childeren in Germany, World Library and Information Congress: 69th IFLA General Conference and Council, 1-9 ağustos 2003, Berlin
- Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
- Mcglohen, M.K. (2004). The Application of a Cognitive Diagnosis Model via an Analysis of a Large-Scale Assessment and a Computerized Adaptive Testing Administration. Yayımlanmamış Doktora Tezi, The University of Texas Austin
- Murphy, K. R. & Davidshofer, C. O. (2001). Psychological testing. (5th ed.). New Jersey: Prentice- Hall,Inc
- Nartgün, Z. (2007). Aynı Puan üzerinden Yapılan Mutlak ve Bağıl Değerlendirme Uygulamalarının Notlarda Farklılık Oluşturup Oluşturmadığına İlişkin Bir İnceleme. Ege Eğitim Dergisi 2007(8), sayı: 1 ss: 19,40
- Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 345–354.
- Waldow, F. (2009). What PISA Did and Did Not Do: Germany after the ‘PISA-shock’, European Educational Research Journal, Sayı: 8 No: 3 ss: 476-483
- Wenmin, Z. (2006). Detecting Differential Item Functioning Using the DINA Model. The University of North Carolina at Greensboro. Unpublished doctorate thesis. Greensboro
Bağıl ve Mutlak Değerlendirme ile DINA Modele Göre Yapılan Sınıflamaların Geçerliğinin Karşılaştırılması / Comparison of Validity of Classification Made According to DINA Model with Criterion-Referenced and Normative Assessment
Abstract
Bu çalışmada, Bilişsel Tanı Modellerinden(BTM) biri olan DINA model ve geleneksel yöntemlerle yapılan sınıflamaların geçerliği incelenmiştir. DINA model, ikili yetenek modellerine benzeyen bir örtük sınıf modelidir. Araştırma, geliştirilen bir testin BTM’ye uygunluk düzeyini belirleme bakımından var olan bir durumu ortaya çıkartmak amacı taşımaktadır. Bu nedenle bu araştırma betimsel bir araştırma olarak değerlendirilebilir. Bunu yanında sınıflama geçerliğine ilişkin farklı teknikleri karşılaştırma olanağı da verdiğinden kuramsal bir araştırma olarak da ele alınabilir. İki yöntem arasında karşılaştırılmaların yapılabilmesi için DINA modelle analiz yapılmaya uygun “eğitimde ölçme ve değerlendirme” dersine ait bir ölçme aracı geliştirilmiştir. Ölçme aracındaki her bir maddeyi doğru cevaplamak için gerekli olan özellikler uzmanlar tarafından belirlenmiş ve madde özellik ilişkisini gösteren Q matrisi uzman uyumu gözetilerek hazırlanmıştır. Mutlak ölçüt kullanılarak yapılan değerlendirme ve DINA model sınıflandırmalarının karşılaştırması sonucunda, öğrenciler hakkında mutlak ölçütle belirlenen geçti kaldı kararlarıyla DINA modelle belirlenen öğrencilerin sahip olduğu özellik sayısı arasında %84.5 oranında bir uyum olduğu gözlenmiştir. Çalışma sonuçları iki yöntemin uyumu kadar, hakkında geçti ya da kaldı kararı verilen öğrencilerin sahip oldukları özellik sayıları bakımından da değerlendirilmelidir. Bu noktada araştırmada dikkat çekilmek istenen önemli bir noktada belirlenen özelliklerden çoğuna sahip olmasına rağmen mutlak değerlendirme içinde hakkında kaldı kararı verilen öğrencilerdir.
References
- Cheng Y.& Chang H.(2007). The Modified Maximum Global Discrimination Index Method for Cognitive Diagnostic Computerized Adaptive Testing. Presented at the CAT and Cognitive Structure Paper Session, June 7
- Cheng, Y. (2010). Improving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method. Educational and Psychological Measurement, 70 (6), 902-913
- de la Torre, J. (2008a). The generalized DINA model framework. Unpublished manuscript. State University of New Jersey.
- de la Torre, J. (2008b). The generalized DINA model. Uluslar arası Psychometric Society toplantısında sunulmuştur, Temmuz, Durham, NH.
- de la Torre, J. (2009a). A cognitive diagnosis model for cognitively-based multiple-choice options. Applied Psychological Measurement, 33, 163–183
- de la Torre, J. (2009b). DINA Model and Parameter Estimation: A Didactic. Journal of Educational and Behavioral Statistics March, Vol. 34, No. 1, ss. 115–130
- de la Torre, J. (2010). The Generalized DINA Model Framework, National Science Foundation Grant tarafından desteklenen araştırma raporu.
- de la Torre, J. , Hong, Y. & Deng, W. (2010). Factors Affecting the Item Parameter Estimation and Classification Accuracy of the DINA Model, Journal of Educational Measurement, Bahar 2010, Sayı 47, No:2, s: 227-249
- de la Torre, J. .& Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika. V69, 3, s. 333-353.
- de la Torre, J. .& Douglas, J. (2008). Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data. Psychometrika. V73, N3, s. 595-624.
- Fagan, T. (2002). Implications of the No Child Left Behind Act for States in the Southwestern Region, Southwest Texas: Educational Development Laboratory, Austin.
- Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
- Haertel, E.H. (1990). Continuous and discrete latent structure models of item response data. Psychometrika 55, 477–494.
- Huebner, A. (2010). An Overview of Recent Developments in Congnitive Diagnostic Computer Adaptive Assessments, Practical Assessment, Research & Evaluation, Vol15, Sayı 3. Ocak
- Huebner, A., Wang, B. & Lee S. (2009). Practical Issues Concerning the Application of the DINA Model to CAT Data, D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.
- Krüger, S. (2003). The PISA-shock and its Consequences: The Future of Libraries for Childeren in Germany, World Library and Information Congress: 69th IFLA General Conference and Council, 1-9 ağustos 2003, Berlin
- Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
- Mcglohen, M.K. (2004). The Application of a Cognitive Diagnosis Model via an Analysis of a Large-Scale Assessment and a Computerized Adaptive Testing Administration. Yayımlanmamış Doktora Tezi, The University of Texas Austin
- Murphy, K. R. & Davidshofer, C. O. (2001). Psychological testing. (5th ed.). New Jersey: Prentice- Hall,Inc
- Nartgün, Z. (2007). Aynı Puan üzerinden Yapılan Mutlak ve Bağıl Değerlendirme Uygulamalarının Notlarda Farklılık Oluşturup Oluşturmadığına İlişkin Bir İnceleme. Ege Eğitim Dergisi 2007(8), sayı: 1 ss: 19,40
- Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 345–354.
- Waldow, F. (2009). What PISA Did and Did Not Do: Germany after the ‘PISA-shock’, European Educational Research Journal, Sayı: 8 No: 3 ss: 476-483
- Wenmin, Z. (2006). Detecting Differential Item Functioning Using the DINA Model. The University of North Carolina at Greensboro. Unpublished doctorate thesis. Greensboro
Details
| Primary Language | English |
|---|---|
| Subjects | Studies on Education |
| Journal Section | Educational Sciences and Sciences of Field Education |
| Authors | |
| Publication Date | April 28, 2014 |
| Published in Issue | Year 2014Volume: 5 Issue: 1 |
Cite
This journal uses a CC BY-NC-SA license.
[email protected] http://www.e-ijer.com Address: Ege University Faculty of Education İzmir/Türkiye