Databases & NetworksMassive lexicon of word relationships279.99For sale: Over 4 MILLION words and phrases, in several languages, and the semantic relationships between them. Read on.
Several years ago, I was doing a project which necessitated finding words that were "related to" one that the user supplied. It wasn't enough to merely import a thesaurus - For instance, if someone typed "lawyer", I didn't want "barrister, attorney, counsel" - I wanted my query to return "divorce, subpoena, accident, lawsuit," etc.
Finding that there was no such data easily available even for sale, I had to create my own.
Finding words was easy: there are many public-domain dictionaries out there and I merely imported them all and condensed them into one. To find the word relationships, I embarked on a 6 month project of constructing algorithms that find words that are commonly associated by frequency on small documents found on the internet, and in large public-domain literature and periodical collections.
The end result is three large SQL tables. The first, "words", contains words in 6 European languages. "related" contains word associations between words using the first of my algorithms, and "associated" contains somewhat looser associations based on the second algorithm.
The algorithms used are fairly simple, but the work required to sift the data required several computers running nonstop for nearly all of the 6 months, using custom-made scripts.
When all the work was done, the project for which I had originally started this adventure ran out of funding. So now I'm selling to you 6 months worth of intense work for less than $300.
What you get here is a Dump of the MySQL database, compressed into a ZIP. When you uncompress it, the resulting text file can be imported into any database management tool to reconstruct the three tables.
I guess the thing I'm trying to express here is: this is not a silly little thesaurus. It's a serious pile of data that has had a lot of work put into it, containing word *relationships* that can't be gleaned from any other source.http://www.scubbly.com/item/43981/
For sale: Over 4 MILLION words and phrases, in several languages, and the semantic relationships between them. Read on.
Several years ago, I was doing a project which necessitated finding words that were "related to" one that the user supplied. It wasn't enough to merely import a thesaurus - For instance, if someone typed "lawyer", I didn't want "barrister, attorney, counsel" - I wanted my query to return "divorce, subpoena, accident, lawsuit," etc.
Finding that there was no such data easily available even for sale, I had to create my own.
Finding words was easy: there are many public-domain dictionaries out there and I merely imported them all and condensed them into one. To find the word relationships, I embarked on a 6 month project of constructing algorithms that find words that are commonly associated by frequency on small documents found on the internet, and in large public-domain literature and periodical collections.
The end result is three large SQL tables. The first, "words", contains words in 6 European languages. "related" contains word associations between words using the first of my algorithms, and "associated" contains somewhat looser associations based on the second algorithm.
The algorithms used are fairly simple, but the work required to sift the data required several computers running nonstop for nearly all of the 6 months, using custom-made scripts.
When all the work was done, the project for which I had originally started this adventure ran out of funding. So now I'm selling to you 6 months worth of intense work for less than $300.
What you get here is a Dump of the MySQL database, compressed into a ZIP. When you uncompress it, the resulting text file can be imported into any database management tool to reconstruct the three tables.
I guess the thing I'm trying to express here is: this is not a silly little thesaurus. It's a serious pile of data that has had a lot of work put into it, containing word *relationships* that can't be gleaned from any other source.
