I was working on OAuth 2.0 specification implementation. For that I wanted to have some utility or logic to validate payload based on JSON schema. I know what everyone thinks lets get package that can get the job done😅. Most cases that is good idea like for production. But doing that we are not actually learning anything or does not know how the package is working. So I thought if I am gonna spend some time in understanding how to use library why not create custom utility that can do the same thing with more flexibility. I had previous experience of working with form generation logic for react project. So I started working on schema validator and I wanted to share how easy it would be for you to implement it.
What is needed for schema validator? Let’s list down things we would need to work our way through it and then jump into implementation.
- Define how we should define our schema.
- Define property value type like what type of data that property / key would contain for example String, Number, Array etc
- Think of schema validator function strategy / algorithm. Here we can think about how we need to validate object with this schema, if user can include custom functions to validate, default validation functions etc.
- Finally optimize it for performance of big schemas.
Now we have some points to work with. Lets tackle each point separately.
1. Schema structure
We can start thinking about how we want to structure our schema. Here is my approach for this
const schama = {
properties: {
firstName: {
type: "String", // type of value
required: true, // Is this value mandatory?
maxLength: 30 // Maximum length for data?
},
lastName: {
type: "String",
required: false,
minLength: 30
},
email: {
type: "Email",
required: true
},
password: {
type: "String",
required: true,
minLength: 8, // min length for data?
maxLength: 48,
maxConsecutiveChars: 2, // How many repeated chars
minUpperCase: 1, // min upper case chars
minNumbers: 1, // min numbers
minSymbols: 1, // min symbols
errorMessage:
"Password is invalid. Should contain min 8 character, max 48 characters, no consecutive characters, atleast one number, atleast one symbol, atleast one uppercase character" // Error msg
}
}
}Let’s break it down to understand what we have defined here
- properties → Schema needs to have properties? why can’t we add it directly at root level. The reason we are doing it like this to handle cases for nested object validation. It would much easier to find nested properties that going through all keys user can add to figure out if it would have any nested values.
- type → type is nothing but what type of data we are expecting to validate. It could be primitive types or custom types if we want to add support for it. Like here in example we have type as “email”.
- required → is value for the field is mandatory?
- maxLength → maximum number of chars accepted
- minLength → minimun number of chars accepted
- maxConsecutiveChars → maximum consecutive number of chars accepted
- minUpperCase → minimum number of upper case chars accepted
- minNumbers → minimum number of numbers accepted
- minSymbols → minimum number of symbols accepted
- errorMessage → custom validation message, if not provided we will assign default validation message (Field Name is not valid).
2. Value type
Now what kind of data we can accept and validate? For simplicity we will just add few primitive types and Email as custom type.
3. Schema validator strategy / algorithm
- As we are saying our schema can accept nested object and can validate nested objects. So naturally we can think of one recursive function to validate nested objects against schema.
- Now we would need validator function for each key / property. As each field can have multiple validations. So here we will have to keep validators array to keep all validation functions for each field. And run one by one to till we validate field or get validation error.
- Ohh now are talking about validation error. For that we would need to decide structure of validation error. Let’s ask user if user would like to get all errors or just return first failed validation. To keep this we will create validation class to hold validation errors related things. And we are taking one more input from user which we will keep it our main SchemaValidator classs.
4. Optimize it for runtime
- So now we have overview on how we will implement it but will we be doing this every time we have to validate schema?
- I was reading on validators and how others have implemented. We can optimize it to have compiled schemas. Meaning we compile schema only once which will have validator functions for each field and we just run validator functions whenever required. Offcource we are just thinking on environments where we start application and it is running for all users. Like Node JS server. But we can kind of tweak it to use to frontend like React. Later on that at the end.
We have blueprint now lets build 🏗
Schema validator class
Lets define our schema validator class. I have added in line comments for us to understand what is happening and what function is doing what. Note this utility was written with typescript. So please ignore typescript stuff. We will write methods in JavaScript to understand.
class SchemaValidator {
// If all field validation errors are required, set it as true. Or else it would break on first validation error.
private allErrors: boolean;// User defined schemas
private schemas: { [key: string]: Schema };// Compiled schemas
private compiledSchemas: { [key: string]: Schema };// Class instance -- for singleton class
private static instance: SchemaValidator;// Private constructor for singleton class
private constructor({
allErrors
}: {
allErrors: boolean
}) {
this.allErrors = allErrors;
this.schemas = {};
this.compiledSchemas = {};
}// static method to get class instance
public static getInstance({
allErrors,
debug = false
}: {
allErrors?: boolean
}): SchemaValidator {
if (!SchemaValidator.instance) {
SchemaValidator.instance = new SchemaValidator({
allErrors: !!allErrors
});
}
return SchemaValidator.instance;
}// helper / setter method to add compiled schemas to validator class
private addSchema(name: string, schema: Schema) {
this.schemas[name] = schema;
}// validate schama against object
public validate(
name: string,
object: { [key: string]: any }
): SchemaValidationError {} /**
* Returns validate function which will take object as parameter to validate
*/
public compile(name: string, _schema?: Schema): Function {
}
}
Now we have defined class its time to work on methods to validate. This is with singleton design pattern, but you can use normal Node js module caching for always exporting same instance of class. For more information about module caching read this. We are accepting input from user in constructor that is why we have this approach. Alternatively it can set from setter function. You have flexibility to choose approach.
Schema validation error class
Validation error class which will hold our validation errors during each schema validations.
class SchemaValidationError {
private errors: SchemaError[];
constructor() {
this.errors = [] as SchemaError[];
}
// add single error to collection
public addError(error: SchemaError) {
this.errors.push(error);
}// add mulitple errors to collection
public addErrors(errors: SchemaError[]) {
this.errors = [...this.errors, ...errors];
}// get all errors
public getErrors(): SchemaError[] {
return this.errors;
}// check if there are any validation errors
public hasSchemaValidationError(): boolean {
return !!this.errors.length;
}
}
Compile method
Here is how we will compile our schemas. Don’t worry about actual validator functions now we will define it at the end.
public compile(name, schema): Function {
if (
(this.schemas[name] && !isObject(schema)) ||
(schema && isObject(schema))
) {
console.log(
`schema was provided to compile function, adding schema to collection, if already exist it will be with new schema`
);
schema && this.addSchema(name, schema);
} else {
console.error(
`compilation of schema(${name}) failed because no schema is not an object, skipping compilation`
);
return () => {};
}// or just use from parameters does not matter :)
const schema = this.schemas[name];if (!schema) {
console.error(
`compilation of schema(${name}) failed because no schema found, skipping compilation`
);// return empty validate function
return () => {};
}// loop through each property to compile single validator function for each field.
for (const property of Object.keys(schema.properties)) {const {
nullable,
required,
minLength,
maxLength,
minLowerCase,
minUpperCase,
minNumbers,
minSymbols,
maxConsecutiveChars,
validator // custom validator function
} = schema.properties[property];let validators = [] as Function[];// Generic function which we will add for every field. Which is just validating type of the value based on schema. Default type is String.
validators.push(validations.isTypeMatches);//Required validation
if (required) {
validators.push(validations.isRequired);
}//minLength validation
if (minLength) {
validators.push(validations.isMinLength);
}//maxLength validation
if (maxLength) {
validators.push(validations.isMaxLength);
}//minLowercase validation
if (minLowerCase) {
validators.push(validations.isMinLowerCase);
}//minUppercase validation
if (minUpperCase) {
validators.push(validations.isMinUpperCase);
}//minUppercase validation
if (minNumbers) {
validators.push(validations.isMinUpperCase);
}//minSymbols validation
if (minSymbols) {
validators.push(validations.isMinSymbols);
}//maxConsecutiveChars validation
if (maxConsecutiveChars) {
validators.push(validations.isMaxConsecutiveCharsExceed);
}//Custom validation function
if (validator && isFunction(validator)) {
console.log(
`Schema has custom validator function, please note custom functions should return true or false as result in any case. True for successful validation, and false for failed validation`
);
validators.push(validator);
}schema.properties[property].validator = (fieldValue) => {
console.log(`Validating for property ${property}`);const isValid = validations.validate(
property,
fieldValue,
validators,
schema.properties[property]
);console.log(`Validation result for property ${property} : ${isValid}`);// If value is valid then we return empty validation msg
if (isValid) {
return "";
}// Return validation msg from schema or default validation msg
return (
schema.properties[property].errorMessage || `${property} is not valid`
);
};
}this.compiledSchemas[name] = schema;
console.log(`compiled schema validation for schema(${name})`);return (objectToValidate) => {
return this.validate(name, objectToValidate);
};
}
Validate method
Here is how we will validate provided object against schemas. Again we will define actual validations at the end. 🙂
public validate(
name,
objectToValidate,
) {
console.log(`validating objectToValidate against schema(${name})`);
let schemaValidationError = new SchemaValidationError();// If objectToValidate is not an object return error
if (!isObject(objectToValidate)) {
schemaValidationError.addError({
field: "payload",
error: `Payload(${name.split(".").pop()}) should be valid objectToValidate`
});
return schemaValidationError;
}// Loop through each property from schema to validate value from objectToValidate against it
for (const property of objectToValidate.keys(this.schemas[name].properties)) {const {
required = false,
type = "String",
validator
} = this.schemas[name].properties[property];// if value is not required we will not run any validation. Offcourse we can change this to if there is value we validate other things if not we return
if (!(property in objectToValidate) && !required) {
continue;
}// Handle nested objects
if (type === "Object") {
const validationResult = this.validate(
`${name}.${property}`,
genericUtils.getValueFromobjectToValidate(objectToValidate, property)
);
if (validationResult.hasSchemaValidationError()) {
schemaValidationError.addErrors(validationResult.getErrors());
}
continue;
}// If validator function exist, validate field
if (validator && isFunction(validator)) {
const validationMsg = validator(
genericUtils.getValueFromobjectToValidate(objectToValidate, property)
);
if (validationMsg) {
schemaValidationError.addError({
field: property,
error: validationMsg
});
if (!this.allErrors) {
break;
}
}
}
}
return schemaValidationError;
}
Now we are done with schemaValidator class 😅 let’s write our validators which are being used in above class.
Validations
validations.js >>>
For simplicity we are using validator library for email validation.
import isEmail from "validator/lib/isEmail";const isTypeMatches = (
value,
type = "String"
) => {
if (type === "Email") {
return isValidEmail(value);
}
return Object.prototype.toString.call(value) === `[object ${type}]`;
};const isBoolean = (value) => isTypeMatches(value, { type: "Boolean" });export const isObject = (value) => isTypeMatches(value, { type: "Object" });const isFunction = (value) => isTypeMatches(value, { type: "Function" });const isString = (value) => isTypeMatches(value);const isNumber = (value) => isTypeMatches(value, { type: "Number"});const isArray = (value) => isTypeMatches(value, { type: "String"});const isUndefined = (value) => isTypeMatches(value, { type: "Undefined"});const isNull = (value) => isTypeMatches(value, { type: "Null" });export const isEmptyObject = (obj: object) =>
JSON.stringify({}) === JSON.stringify(obj);// Using validator library to validate email
const isValidEmail = (email) => {
return isString(email) && isEmail(email);
};// Required validation
const isRequired = (value) => {
if (isBoolean(value)) {
return value;
}
if (isObject(value)) {
return !isEmptyObject(value);
}// Here we are considering number should be greater than 0 if it is required
if (isNumber(value)) {
return value > 0;
}
return !!value;
};const isMinLength = (value, { minLength = 0 }) => {
if (isArray(value) || isString(value)) {
return value.length >= minLength;
}
if (isNumber(value)) {
return String(value).length >= minLength;
}
console.log(
`${value} is of type which we cannot determine minLength. Please check validation schema`
);
return false;
};const isMaxLength = (value, { maxLength = 0 }) => {
if (isArray(value) || isString(value)) {
return value.length <= maxLength;
}
if (isNumber(value)) {
return String(value).length <= maxLength;
}
console.log(
`${value} is of type which we cannot determine maxLength. Please check validation schema`
);
return false;
};const isMinLowerCase = (
value,
{ minLowerCase = 0 }
) => {
if (isString(value)) {
return ((value || "").match(/[a-z]/g) || []).length >= minLowerCase;
}
console.log(
`${value} is of type which we cannot find lower cases. Please check validation schema`
);
return false;
};const isMinUpperCase = (
value,
{ minUpperCase = 0 }
) => {
const log = new Logger(`${isMinUpperCase.name}`);
if (isString(value)) {
return ((value || "").match(/[A-Z]/g) || []).length >= minUpperCase;
}
console.log(
`${value} is of type which we cannot find lower cases. Please check validation schema`
);
return false;
};const isMinNumbers = (value, { minNumbers = 0 }) => {
const log = new Logger(`${isMinNumbers.name}`);
if (isString(value) || isNumber(value)) {
return (
((value || String(value || "")).match(/\d/g) || []).length >= minNumbers
);
}
console.log(
`${value} is of type which we cannot find numbers. Please check validation schema`
);
return false;
};const isMinSymbols = (value, { minSymbols = 0 }) => {
const log = new Logger(`${isMinSymbols.name}`);
if (isString(value)) {
return (value.match(/[^\p{L}\d\s]/u) || []).length >= minSymbols;
}
console.log(
`${value} is of type which we cannot find symbols. Please check validation schema`
);
return false;
};const isMaxConsecutiveCharsExceed = (
value,
{ maxConsecutiveChars = 0 }
) => {
const log = new Logger(`${isMinSymbols.name}`);
if (isString(value)) {
const charCountMap: GenericObject = {};
const valueArray[] = Array.from(value);
valueArray.forEach(char => {
char = String(char).toLocaleLowerCase();
const val = charCountMap[char];
if (val) {
charCountMap[char] += 1;
} else {
charCountMap[char] = 1;
}
});
return !Object.values(charCountMap).find(
charCount => charCount > maxConsecutiveChars
);
}
console.log(
`${value} is of type which we cannot find consecutive characters. Please check validation schema`
);
return false;
};export const validate = (
propertyKey,
value,
validators,
validationRule
) => {
const log = new Logger(`${fileName}.${validate.name}`);
console.log(`validating for ${propertyKey} with value as ${value}`);
let isValid = true;
for (const validationFunction of validators) {
if (isFunction(validationFunction)) {
isValid = validationFunction(value, validationRule);
console.log(
`validation function (${validationFunction.name}) result: ${isValid}`
);
if (!isValid) {
break;
}
} else {
console.log(
`${validationFunction} is not a function, skipping this validation function`
);
}
}
console.log(
`validation complete, value for ${propertyKey} is ${
isValid ? "valid" : "invalid"
}`
);
return isValid;
};export default {
isBoolean,
isObject,
isString,
isNumber,
isEmptyObject,
isTypeMatches,
isRequired,
isMinLength,
isMaxLength,
isMinLowerCase,
isMinUpperCase,
isMinNumbers,
isMinSymbols,
isMaxConsecutiveCharsExceed,
validate,
};
we have already taken care of optimization for usage as we are compiling the validators along with JSON schema. This is good idea as it will run only once on app start. But what about single page applications? wouldn’t that cause issues, ideally yes and no. It depends on how big your schema can get but better idea like React applications would add validator.js during build time. What I mean is compile all your JSON schema’s during runtime and create valudator.js with all validator functions after compilation. And use that as reference in project. It add more work but it would help with big schema validations. 😀
Thats it now we have a framework to work with. Now we can add any custom validation to our schema validator. If you do it your self by understanding it give you experince with designing small utilities.
You can follow source code with some extra validations here.
Conclusion: you can write your own custom JSON schema validator. Which is more flexible and you have hands on to debug issues if any 😉.
Note: This is very basic example here for demo. This is not production ready code.
Good to know: If there is need to use standard library like in production environment by all means use the standard libraries as those are well maintained. Only thing is you still need to spend some time to understand how to use it, but still if you know how library works then it is fine to use library. If you don’t know how library is working then dig a little deeper and see how it works. It will help you understand issues from production if there are any.
Good Luck 🙂