The top reasons why knowledge base size matters

A key to useful and accurate open source compliance and security is a powerful software analysis tool, relying on a…

A key to useful and accurate open source compliance and security is a powerful software analysis tool, relying on a comprehensive knowledge base. The FossID knowledge base offers a commercial representation of the latest innovation in open source software, and references billions of open source projects, files, snippets, and vulnerabilities.


A fully automated approach building a knowledge base

Software is eating the world. It has become ubiquitous in everything from home appliances to intelligent cars and an overwhelming amount of software is open source based. Open source multiplies the development speed significantly, and drives standardization of non-differentiating functionality, which bolsters differentiation and niche innovation.

Open source software grows exponentially, and new projects are created by the minute. There are trillions of lines of open source code and billions of code snippets and they all come with licenses and license obligations that need to be honored.

But how is it even possible to keep track of this vast amount of code, and be updated with the latest security vulnerabilities? The answer is automated data collection.

To keep up with the ever-changing open source landscape, FossID has developed an automated mechanism that looks at the data in almost a hundred public source code repositories and quarries and stores it as cryptographic hashes in the FossID Knowledge Base. From sources like GitHub, Maven and Googlesource, to Gnu, Kernel and Pypi, FossID represents every vertical and every use case imaginable. And we keep growing our list of sources, even from StackOverflow and other user contribution forums.

Automated data collection - FossID knowledge base


A new database technology bespoke for speed and compression

With software being the new black, traditional methods of building a knowledge base were simply insufficient in compression capabilities and performance. That is why FossID built its own from scratch, and since its creation it has matured in several generations and currently has four pending patents attached to it.

Great compression rate brings light-weight deployments

FossID packs an equivalent of 2PB of open source reference information into its knowledge base, and compresses it in a proprietary, patented, manner down to 6TB. The knowledge base format also allows for very fast scan rate performance of 70files/s on average.

“We know what you copy-pasted last summer”

Open source compliance and security vulnerability detection is mostly a risk management exercise. On one hand, you want to be compliant with applicable licenses for all source code included in your products and services and avoid security vulnerabilities. On the other hand, you want to allow your developers the flexibility of using both whole components and re-using files or partial code snippets originating from open source projects. FossID detects and identifies snippets down to six lines of code.

Revolutionary security vulnerability detection down to snippet level

FossID attacks open source security vulnerabilities on a whole new level. Using sources like the National Vulnerability Database (NVD), Bugzilla, Android Security, and others we detect not only the component or file that is known to have introduced security vulnerabilities, but the actual lines of code. This makes the analysis easier, since we for each snippet only list the relevant vulnerability and don’t overwhelm the user with unprecise results lists and false positives. For each hit we indicate source of origin, version number, license name and point to additional information and possible remediation.

Frequent updates for access to the latest open source evolution

For regular deployments where the customer uses a knowledge base in the FossID cloud, it is always up to date with the latest evolution in the open source world, giving customers access to the latest updates as they happen. For “offline” deployments, where customers host the knowledge base on their own premise, regular updates are either downloaded over the air, or sent physically on smaller servers.

database technology


Data sheet describing the FossID knowledge base of open source software and security vulnerabilities.



Other Articles relevant