MADLaT Conference - Stephen Downes
Stephen Downes is pulling together a wide range of resources and work over the last several years. He has previously posted articles, etc. (resource profiles, edu_rss, dlorn, etc.). I attended a session at MADLaT today where he presented on the topic of “Projecting Quality” – based largely on his work in resource profiles. The challenge is complex: trying to determine the quality of unrated learning object by evaluating the nature of other similar objects and other similar users. Think of a dating.com meets the educational world.
Here’s a summary of his notes (I’ll post an updated link once he posts his own files).
Selection of learning objects – based on metadata
Filtering/sorting according to rated quality
Problem: “takes too long to create customer learning”…want to reduce development time for learning to zero. Want to find learning objects with search results of one.
Sifter Organizing Project
Learning resources (formerly known as learning objects) – want to make metadata available for harvesting. LO metadata is descriptive of resource…evaluation of LO is descriptive of resource.
MERLOT
- peer review process
- materials triaged to presort for quality
- pointer, not actual object posted
LORI
- members browse collection of LO
- review is an aggregate of member reviews
Issues:
- Merlot – peer review is too slow – bottleneck
- MERLOT and LORI are centralized
- Single criteria – media requires different criteria
- Results are a single aggregation, but different types of users have different criteria (i.e. student will assess a resource differently than a designer)
What we wanted:
- a method for determining how a learning resource will be appropriate for a certain use when it hasn’t been seen/reviewed (i.e. projecting quality – trying to predict value)
- system to collect and distribute learning resource evaluation metadata (associates quality with known properties – author, publisher, etc.)
Attention turned to “recommender systems” (collaborative filtering use database about user preference to predict additional resources)
The idea is that associations are mapped:
- user profile
- resource profile
- previous evaluations of other resources
Firefly: one of earliest recommender systems on the web
- user created profile
- user profile stored in “Passport”
Launch.com – Yahoo! – online music service – you rate selections. Detailed personal profiling available
Match.com – dating site, create user profile, adds personality tests
Our methodology:
- multidimensional evaluation of LO’s
- Build quality criteria based on metadata ratings
- Use model to assign a quality value to unrelated LO’s
- Update object’s profile according to its history of use
Rethinking LO Metadata – existing LO metadata conceptions is insufficient:
- Getting the description right
- Problem of trust
- Multiple descriptions
- New types of metata
Concept of resource profiles developed to allow use of evaluation metadata.
http://www.downes.ca/files/resource_profiles.htm
- multiple vocabularies – for different types of objects
- multiple authors – content author , publisher
- distributed metadata – reaction to centralized system. Evaluation of LO involves third party
- metadata models
- analogy – personal profiles – additional resources are used to evaluate a resource (just like people are evaluated by more than their resumes)
Three types of metadata:
- First person – bibliographic, technical, rights – created by content author
- Second person – education, sequence, interaction – created by content user in process of use
- Third person – evaluation, classification – done by third parties
Our (NRC) approach
- Quality evaluator using LO type-specific evaluation criteria
- information according to eight groups of LO users
- weighted global rating
- user-tailored rating ratings schemes must be normalized to include the habits of individual raters (i.e. someone who rates only 1-5 on a scale of 1-10 will have their responses weighted to reflect their own habits)
- Combination of subjective quality values are purposefully fuzzy –
- Representing evaluation data: as an XML file available for harvesting along side LO metadata
User profile: user description and automatically collected
Content is filtered based on content similarities. Collaborative filtering – used when ratings of LO’s are available, no metadata (projections into the “future” derived from user, object)
LO prediction:
- Calculate objects similarity to others
- User similarity
- Predict quality value of the unrated LO (based on LO and user properties)
Questions after the session:
Question: nature of user profile – are they static, can they be updated?
Answer: FOAF – an xml file maintained by the person in question, which is made available for questioning. Individual (self evaluation) is not complete. Other evaluations are needed to improve accuracy of evaluation. As a person uses a system, the sorts of things they do should be captured and made available for systems by harvesting (though this does raise issues of privacy). Multi-dimensional descriptions and their interests as determined by their actions are developing. We should treat personal information in the same way we treat copyright information. A similar principle is needed for description of use of individual profiles. Ownership and control of this information has to reside with the individual in question.
Question: How do we keep a record of where we’ve been in LOs?
Answer: Search should not be an explicit task. It should occur in the background while we are completing our tasks. The system should go to resource libraries and begin searching for and presenting resources to the end user. This is a long term vision. It’ll take another 20 years. The design of what constitutes a class becomes less and less content based, but more task/activity (doing something). The information should be delivered automatically…and resource evaluation is an important part of this. Wide spread evaluation with many points of view need to be dynamic.
Posted by gsiemens at May 7, 2004 5:49 PM