EduDDI at IASSIST Tampere May 2009

EduDDI – an application of DDI 3.x for large
scale assessments in education
IASSIST/IFDO 2009 Tampere, Finland
May 26th-May 29th, 2009
Martin Mechtel, Dr. Michel Knigge
Outline
• IQB – Institute for Educational Progress
• Expected benefits of meta data standardisation
• Efforts so far (drawbacks)
• Introducing DDI 3.x (bridges)
• Road map of EduDDI
(sorry no software demo)
IQB – Institute for Progress in Education
•
Founded 2004 by the Bundesländer (Federal States); hosted by
Humboldt-University at Berlin, Principal Prof. Dr. Olaf Köller
•
Trigger: TIMSS 1995/PISA 2000; frequent large scale assessments of
students competencies take place on a regular basis since then
•
Mission: Test educational standards for Germany
•
Staff: Meanwhile 40 plus 40 part time (students)
•
Budget: About 4.5 Mio € per year, plus extra projects (e.g. FDZ –
research data center)
•
My role: Responsible for IT services (infrastructure, development);
IT-staff: 3 plus 4 part time collegues (most infrastructure is supported
by Humboldt-University)
•
Dr. Michel Knigge: Head of FDZ
Expected benefits of meta data standardisation
Obvious:
•
Interoperability: not only "meta data" but datasets too
•
Exchange of items/tasks easily, support of re-analyses, nice
presentation of data to the public etc.
•
Thinking: when I have a standard then I have tools which do
the work for me
Side effects:
•
Installation of structures, work flows and vocabularies
•
Dissemination of best practises (statistical methods, software for
analyses, reporting culture, access policies)
Efforts so far (drawbacks) – specific in education
1
•
Item appearance is crucial for interpretation of results (line breaks, font,
size etc.)
•
Item structure is complex (task: stimulus, sub tasks, items for scoring,
variables for analyses)
Efforts so far (drawbacks) – specific in education
Vogonische Lyrik
Task M042
Grupp, ich beschwöre dich, mein punzig Turteldrom
Und drängel reifig mich mit krinklen Bindelwördeln
Denn sonst werd ich dich rändern in deinen
Stimulus
Gobberwarzen
Mit meinem Börgelkranze, wart´s nur ab!
•
Aufgabe 1
•
item appearance is crucial for interpretation of results (line breaks, font,
size etc.)
item structure is complex (task: stimulus, sub tasks, items for scoring,
variables for analyses)
Welches Sprichwort passt am ehesten zu der
Aussage des Gedichts?
Kreuze die richtige Antwort an.
 Reisende soll man nicht aufhalten
 Liebe macht blind
 Ohne Fleiß keinen Preis
 Nachts sind alle Katzen grau.
Subtask A
Item 01
Variable a
An welcher Zeile ist das erkennbar (Zeilennummer):
Item 02
Variable a
Aufgabe 2
Nenne zwei Klangfiguren in diesem Gedicht:
Subtask B
Item 03
Variable a
Variable b
Efforts so far (drawbacks) – specific in education
1
•
Item appearance is crucial for interpretation of results (line breaks, font,
size etc.)
•
Item structure is complex (task: stimulus, sub tasks, items for scoring,
variables for analyses)
2
•
Complex study design to estimate the level of a certain competence
although the test person responds simply to one or two items of the
scale
•
Item position is important for interpretation
Efforts so far (drawbacks) – specific in education
test design (example)
booklets
timeline
total 167 Min
Efforts so far (drawbacks) – specific in education
1
•
Item appearance is crucial for interpretation of results (line breaks, font,
size etc.)
•
Item structure is complex (task: stimulus, sub tasks, items for scoring,
variables for analyses)
2
•
Complex study design to estimate the level of a certain competence
although the test person responds simply to one or two items of the
scale
•
Item position is important for interpretation
3
•
Mostly small institutions involved in assessment (EMSE –
Bundesländer; isolated chairs in research); like simple procedures
•
Mostly short-term projects; no pay-back of investment in
comprehensive approaches
development of a standard failed
Introducing DDI 3.x (bridges)
Vogonische Lyrik
Task M042
Grupp, ich beschwöre dich, mein punzig Turteldrom
Und drängel reifig mich mit krinklen Bindelwördeln
Denn sonst werd ich dich rändern in deinen
Stimulus
Gobberwarzen
Mit meinem Börgelkranze, wart´s nur ab!
But now: we HAVE a standard!
Aufgabe 1
Welches Sprichwort passt am ehesten zu der
Aussage des Gedichts?
Kreuze die richtige Antwort an.
 Reisende soll man nicht aufhalten
 Liebe macht blind
 Ohne Fleiß keinen Preis
 Nachts sind alle Katzen grau.
Subtask A
Item 01
Variable a
An welcher Zeile ist das erkennbar (Zeilennummer):
Item 02
Variable a
Aufgabe 2
Nenne zwei Klangfiguren in diesem Gedicht:
Subtask B
Item 03
Variable a
Variable b
"EduDDI"
Introducing DDI 3.x (bridges)
•
DDI can do: specific structure, other material, scales
•
DDI should do (3.2?): usage of Genericode to attach any property to
any XML item (inline storage of data to get a question bank)
•
Working with DDI expands the community (by YOU)
•
DDI is the cheese for the mouse: possible partners are soon convinced
and support the project
Other fortunate conditions:
•
New high stake/large scale project looks for technology: NEPS National
Educational Panel Study
•
In Germany, a sufficient number of large studies wait for further
analyses (FDZ)
Road map of EduDDI
Project/initiative members: DIPF Frankfurt/M., TestDaF Hagen, IEA-DPC
Hamburg, IQB+FDZ, ifib Bremen, University of Bamberg (NEPS)
Near future
•
Discussion about meta data catalogues (two groups: item oriented and
dataset oriented)
•
Discussion about technology/architecture (inside institutions:
ASP/VB.Net, MS SQL Server; offline tools: java)
•
Acquirement of more support (project proposals, more institutions)
Next year
•
Definition of the DDI implementation
•
Development of first tools and interfaces
•
Launch of research studies to underpin the assumptions and to bring
the project to a broad community
telefon +49[30]2093-5335
telefax +49[30]2093-5336
[email protected]
www.IQB.hu-berlin.de
postal address
Humboldt-Universität zu Berlin
Unter den Linden 6
10099 Berlin
Location
Luisenstr. 56
10117 Berlin