Generic Parser

coders#2

Meet the Team

1. Support at least one of XML or JSON:

2. Ensure Error Handling and Validation:  

3. Provide Flexibility and Customization:  

4. Map the JSON structure to final tables

5. Ability to handle up to 3 levels of nesting

Problem statement

We have come up with 2 approaches to implement generic parser for JSON

• Using an open source library
• Using JS native libraries

Purpose of using 2 different ways is to showcase different algorithms for parsing and compare their run time.

These 2 approaches also differ in the way input is given to the application.

Approach I

Using Native JS Library

Clients : SBI, Kotak, Canara

End-User: Sales team, Engineering team

APIs introduced:

/register
input format:

identifier: string

jsonFile: jsonFile

/parse
input format:

identifier: string

jsonFile: json file

Flow:

Register

• Clients will call register API with identifier string and  sample.json file corresponding to each client, using an API <https://localhost:3000/register>
• This will add a entry in backend and store the json structure corresponding to each client
• In this way we have a map where each key corresponds to each client and value corresponds to json structure
• After this operation we will also create a table corresponding to each client schema

Parse

• For Parsing, end user will call <https://localhost:3000/parse>
• Passed input will be verified against predefined structures in the above map
• If there is any difference in structure then all mismatched fields are returned.
• If there is a match then file is inserted into the DB

Approach II

Techstack : Nodejs, MongoDB

API: http://localhost:5002/api/parse

Input : can be any JSON with name of source

Output:  

success : parsed JSON

Error: Validation Errors/parsing Errors

Implementation Idea:

• Create a meta data file which is configurable and have the name of file as type of data.
• When Api request comes , based on the input type, the meta data file will be matched and json schema will be validated. If there are any validation errors, it throws an error.
• For the JSON shema validation, we used ajv library
• If the validation is success 
  • create the mongodb schema dynamically using json schema and taking table name from meta data.
  • we are converting json schema to mongo schema using convert-json-schema-to-mongoose.
• Then we perform the db query which is mentioned in meta data.
• We send the parsed json data as response. 

Thank You!

Questions?

UI

API

https://drive.google.com/file/d/1ATGYHfB4wmGeW8bgCKpmEK2sqmJZSjPv/view?usp=sharing

https://drive.google.com/file/d/1zhp51FaKu7UAY3B_7B2NnF_vUpXWD9V8/view?usp=sharing