Show simple item record Nakandala, S Withana, SD Kumarasiri, D Jayawardena, H Bandara, HMND Perera, S Marru, S Pamidighantam, S 2015-08-03T10:14:16Z 2015-08-03T10:14:16Z 2015-08-03
dc.description.abstract Modern scientific experiments generate vast volumes of data which are hard to keep track of. Consequently, scientists find it difficult to reuse and share these data sets. We address this problem by developing a schema-independent data cataloging framework for efficient management of scientific data. The proposed solution consists of an agent which automatically identifies new data products and extract metadata from them, as well as a server which indexes the metadata using a NoSQL database and provides a REST API for querying, sharing, and reusing the data sets. The novelty of our solution lies in the pluggable metadata extraction logic, extensible data product generation monitors, use of a NoSQL database, and the ability to dynamically add new metadata fields. The use of Apache Solr as the backend database enables the proposed solution to index and search data products much fatser than a solution based on relational databases. For example, our Apache Solr based implementation can resolve full text, sub-string, prefix, and suffix queries 91% - 99% faster than a MySQL-based implementation. en_US
dc.description.sponsorship IEEE IEEE Sri Lanka Section Robotics and Automation Section Chapter, IEEE Sri Lanka Section en_US
dc.language.iso en en_US
dc.subject indexing; metadata catalog; scientific data management en_US
dc.title Schema-Independent Scientific Data Cataloging Framework en_US
dc.type Conference-Full-text en_US
dc.identifier.faculty Engineering en_US
dc.identifier.department Dept. of Computer Science and Engineering University of Moratuwa Katubedda, Sri Lanka en_US
dc.identifier.department WSO2 Inc. Colombo, Sri Lanka en_US
dc.identifier.department Indiana University Bloomington, USA en_US
dc.identifier.year 2015 en_US
dc.identifier.conference MERCon 2015 Moratuwa Engineering Research Conference en_US u en_US
dc.identifier.pgnos p 71 en_US

Files in this item

This item appears in the following Collection(s)

Show simple item record