{"id":109,"date":"2017-07-04T21:34:07","date_gmt":"2017-07-04T21:34:07","guid":{"rendered":"http:\/\/eurekadata.net\/?page_id=109"},"modified":"2017-10-05T16:55:27","modified_gmt":"2017-10-05T16:55:27","slug":"shakespeare-example","status":"publish","type":"page","link":"https:\/\/eurekadata.net\/index.php\/shakespeare-example\/","title":{"rendered":"Shakespeare Example"},"content":{"rendered":"

 <\/p>\n

In this example we’ll focus on the first sonnet of the William Shakespeare<\/a>. \u00a0The six rules and the priority by order will lex the corpus into six separate Eureka spaces. \u00a0Note that the\u00a0catch all<\/i>\u00a0rule\u00a0\u2018.\u2019<\/i>\u00a0has lowest priority and though it would otherwise capture all remaining single characters it will remain\u00a0empty.<\/p>\n

priority_regex_name_table<\/h2>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
Priority<\/th>Regular Expression Rule<\/th>Space<\/th>\n<\/tr>\n<\/thead>\n
1<\/td>
([A-Za-z]+[0-9]*)+<\/pre><\/td>
word<\/span><\/td>\n<\/tr>\n
2<\/td>
[+-~]?[0-9]+([\\\\.][0-9]*)*<\/pre><\/td>
num<\/span><\/td>\n<\/tr>\n
3<\/td>
(([ \\t]+[:;\/\\.,!@#$%&*()_\\+\\-={}\\^\\[\\]'()\"(<>]*)+)<\/pre><\/td>
white<\/span><\/td>\n<\/tr>\n
4<\/td>
(([:;\/\\.,!@#$%&*()_\\+\\-={}\\^\\[\\]')\"\\(<>]+[ \\t]*)+)<\/pre><\/td>
punct<\/span><\/td>\n<\/tr>\n
5<\/td>
\\n\\n?<\/pre><\/td>
para<\/span><\/td>\n<\/tr>\n
6<\/td>
.<\/pre><\/td>
any<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

And the resulting lexing, color coded to the above would look something like:<\/p>\n

1609<\/span>\u2424
\n<\/span>THE<\/span>\u00a0<\/span>SONNETS<\/span>\u2424
\n<\/span>by<\/span>\u00a0<\/span>William<\/span>\u00a0<\/span>Shakespeare<\/span>\u2424
\n<\/span>\u2424
\n<\/span>\u00a0<\/span>1<\/span>\u2424
\n<\/span>\u00a0<\/span>From<\/span>\u00a0<\/span>fairest<\/span>\u00a0<\/span>creatures<\/span>\u00a0<\/span>we<\/span>\u00a0<\/span>desire<\/span>\u00a0<\/span>increase<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>That<\/span>\u00a0<\/span>thereby<\/span>\u00a0<\/span>beauty<\/span>‘<\/span>s<\/span>\u00a0<\/span>rose<\/span>\u00a0<\/span>might<\/span>\u00a0<\/span>never<\/span>\u00a0<\/span>die<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>But<\/span>\u00a0<\/span>as<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>riper<\/span>\u00a0<\/span>should<\/span>\u00a0<\/span>by<\/span>\u00a0<\/span>time<\/span>\u00a0<\/span>decease<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>His<\/span>\u00a0<\/span>tender<\/span>\u00a0<\/span>heir<\/span>\u00a0<\/span>might<\/span>\u00a0<\/span>bear<\/span>\u00a0<\/span>his<\/span>\u00a0<\/span>anamory<\/span>:<\/span>\u2424
\n<\/span>\u00a0<\/span>But<\/span>\u00a0<\/span>thou<\/span>\u00a0<\/span>contracted<\/span>\u00a0<\/span>to<\/span>\u00a0<\/span>thine<\/span>\u00a0<\/span>own<\/span>\u00a0<\/span>bright<\/span>\u00a0<\/span>eyes<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>Feed<\/span>‘<\/span>st<\/span>\u00a0<\/span>thy<\/span>\u00a0<\/span>light<\/span>‘<\/span>s<\/span>\u00a0<\/span>flame<\/span>\u00a0<\/span>with<\/span>\u00a0<\/span>self<\/span>–<\/span>substantial<\/span>\u00a0<\/span>fuel<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>Making<\/span>\u00a0<\/span>a<\/span>\u00a0<\/span>famine<\/span>\u00a0<\/span>where<\/span>\u00a0<\/span>abundance<\/span>\u00a0<\/span>lies<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>Thy<\/span>\u00a0<\/span>self<\/span>\u00a0<\/span>thy<\/span>\u00a0<\/span>foe<\/span>, <\/span>to<\/span>\u00a0<\/span>thy<\/span>\u00a0<\/span>sweet<\/span>\u00a0<\/span>self<\/span>\u00a0<\/span>too<\/span>\u00a0<\/span>cruel<\/span>:<\/span>\u2424
\n<\/span>\u00a0<\/span>Thou<\/span>\u00a0<\/span>that<\/span>\u00a0<\/span>art<\/span>\u00a0<\/span>now<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>world<\/span>‘<\/span>s<\/span>\u00a0<\/span>fresh<\/span>\u00a0<\/span>ornament<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>And<\/span>\u00a0<\/span>only<\/span>\u00a0<\/span>herald<\/span>\u00a0<\/span>to<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>gaudy<\/span>\u00a0<\/span>spring<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>Within<\/span>\u00a0<\/span>thine<\/span>\u00a0<\/span>own<\/span>\u00a0<\/span>bud<\/span>\u00a0<\/span>buriest<\/span>\u00a0<\/span>thy<\/span>\u00a0<\/span>content<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>And<\/span>\u00a0<\/span>tender<\/span>\u00a0<\/span>churl<\/span>\u00a0<\/span>mak<\/span>‘<\/span>st<\/span>\u00a0<\/span>waste<\/span>\u00a0<\/span>in<\/span>\u00a0<\/span>niggarding<\/span>:<\/span>\u2424
\n<\/span>\u00a0<\/span>Pity<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>world<\/span>, <\/span>or<\/span>\u00a0<\/span>else<\/span>\u00a0<\/span>this<\/span>\u00a0<\/span>glutton<\/span>\u00a0<\/span>be<\/span>,<\/span>\u2424
\n<\/span>\u00a0<\/span>To<\/span>\u00a0<\/span>eat<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>world<\/span>‘<\/span>s<\/span>\u00a0<\/span>due<\/span>, <\/span>by<\/span>\u00a0<\/span>the<\/span>\u00a0<\/span>grave<\/span>\u00a0<\/span>and<\/span>\u00a0<\/span>thee<\/span>.<\/span>\u2424
\n<\/span><\/p>\n

Internally the corpus would be stored in the six spaces with inter-space join tables. \u00a0Queries could search for consecutive tokens in one space, matching specific pattern of spaces or\u00a0any<\/em>.<\/p>\n

word:token1 [\u2295[ .*:.*]+ \u2295 word:tokenYi]*<\/pre>\n
A variation<\/p>\n
word:PC<\/strong><\/em>(token1) [\u2295[ .*:.*]+ \u2295 word:tokenYi]*<\/pre>\n
where PC<\/strong><\/em> would permute and include all existing capitalization of token1 in word space, or<\/p>\n
word:token1 [\u2295[ white:\"\\t\"]+ \u2295 word:tokenYi]*<\/pre>\n
where the white space separator(s) are explicitly constrained to a single tab.<\/p>\n
Indexing fully every letter position in every token, indexing every token (no normalization or dropping of stop-words) and storing all frequency statistics from letter frequencies to the count of unique frequencies present in each space costs only a small multiple (i.e. <3<\/strong> times) of the original corpus size even in this small case. \u00a0As we scale documents up to achieve statistical significance this ratio trends toward one.<\/p>\n
Just as an emphasis on the small nature of the this corpus, and also the Eureka data approach for storing all relevant meta data directly, the word<\/span> space the count of frequencies are displayed in the following table.<\/p>\n
Note that due to the minuscule size this space is very sparse and the numeric progressions are likewise discontinuous:<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
Count of words sharing \u03bb <\/th>Frequency(\u03bb)<\/th>\n<\/tr>\n<\/thead>\n
1<\/td>6<\/td>\n<\/tr>\n
2<\/td>4<\/td>\n<\/tr>\n
4<\/td>3<\/td>\n<\/tr>\n
7<\/td>2<\/td>\n<\/tr>\n
78<\/td>1<\/td>\n<\/tr>\n
92 (total)<\/td><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nFor words in Shakespeare's first sonnet, the distribution of frequencies shows that only one word has six occurrences, two words have four occurrences, four words have three occurrences and a whopping seventy-eight words are only present one time in this short text.  Hows that for diversity!<\/span>\n\n
As the size of the corpus increases the relatively costly insertions of unique terms into the terms table occur and all tables benefit from inherent statistics. \u00a0This has the effect of improving overall per corpus efficiency, but even in the case of our minuscule corpus we are looking reasonably good.<\/p>\n\n\n\n\n\t\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
Space<\/th>Eureka Internal Size<\/th>Original Size (in bytes)<\/th>Ratio of Sizes<\/th>\n<\/tr>\n<\/thead>\n
total<\/th>1.9 kbytes<\/th>711<\/th>2.6<\/th>\n<\/tr>\n<\/tfoot>\n
word<\/span><\/td>1.4 kbytes<\/td>510<\/td>2.9<\/td>\n<\/tr>\n
num<\/span><\/td>59 bytes<\/td>5<\/td>12<\/td>\n<\/tr>\n
white<\/span><\/td>252 bytes<\/td>145<\/td>1.7<\/td>\n<\/tr>\n
punct<\/span><\/td>146 bytes<\/td>28<\/td>5.4<\/td>\n<\/tr>\n
para<\/span><\/td>90 bytes<\/td>24<\/td>3.7<\/td>\n<\/tr>\n
any<\/span><\/td>0<\/td>0<\/td>0\/0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n
Conclusion<\/h3>\n
Eureka is designed around direct joining, hence utilizing the concept of multiple spaces result in very natural access absent performance degradation. \u00a0The demarkation of genuinely<\/em>\u00a0different\u00a0kinds of data will strongly enhance performance as\u00a0<\/em>each space exhibits disparate frequencies, limited tokens and natural association. \u00a0The spacial definition is a natural deconvolution of the original data reducing the overall size and entropy hence increasing performance.<\/p>\n
Though Eureka is general in that it more about discovering structure than imposing structure on a corpus it is wise for the convenience of access and performance to consider the definitions of token rules and priorities as they serve the dual purpose of defining your data and optimizing access.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
  In this example we’ll focus on the first sonnet of the William Shakespeare. \u00a0The six rules and the priority by order will lex the corpus into six separate Eureka spaces. \u00a0Note that the\u00a0catch all\u00a0rule\u00a0\u2018.\u2019\u00a0has lowest priority and though it would otherwise capture all remaining single characters it will remain\u00a0empty. And the resulting lexing, color … Continue reading Shakespeare Example<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/pages\/109"}],"collection":[{"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/comments?post=109"}],"version-history":[{"count":62,"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/pages\/109\/revisions"}],"predecessor-version":[{"id":401,"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/pages\/109\/revisions\/401"}],"wp:attachment":[{"href":"https:\/\/eurekadata.net\/index.php\/wp-json\/wp\/v2\/media?parent=109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}