Many business leaders recognize AI's ability to transform their companies, but are unable to facilitate that transition as they are hindered by AI talent shortages and related expenses / risks. Tech leaders on the other hand have tried to address this by delivering AutoML, MLOps, Low Code and No Code solutions. While all these innovations are continuing to offer value, the best solutions in industry and competitions (e.g. Kaggle) often incorporate human expertise and ingenuity (not AutoML).
Furthermore, the genesis and the biggest breakthroughs in ML have mostly come from processing large volumes homogenous data for applications such as Computer Vision (Image and Video), Speech Recognition and Natural Language Processing. Other advances have come from areas such as Gaming and Robotics. Machine Learning for business on the other hand deals with mostly heterogeneous data with varying volumes. and has been adopting algos, processes and even people trained to solved problems in other domains.
Semantic Brain is addressing the algo, process and people gaps identified above by launching BizML - a combination of Data, Software and Professional / Integration Services optimized to help businesses significantly improve their profitability by leveraging AI in order to facilitate forecasting and optimization.
We typically optimize Finance and Marketing functions and deliver the following benefits
Improve accuracy of financial predictions and marketing optimizations by 5% to 20%, if accuracy is below 80% (conservative estimate, but depends on problem and availability of data)
For example, if current solution accuracy is 60% our approach will typically increase accuracy from 63% to 72%
Reduce recommendation error rate of by 10% to 50%, when accuracy is above 80%
For example, if current solution accuracy is 90% our approach will typically increase accuracy from 91% to 95%
Reduce ML computing needs by more than 70%
Reduce AI/ML development time up to 50%
Substantial staffing cost savings (more than $250K)
Substantial project risk reduction
What is BizML's high level approach?
Semantic Brain's philosophy can be simply stated as
Human + AI > Human or AI
In order to achieve the best possible outcome we work directly with the stakeholders, including the Business Experts / Specialists and Users.
When engaging Business Experts, our goal is to refine, validate and embed their knowledge utilizing BizML. Business Specialists help identify relevant Data, Features and Business Rules that augment BizML's own Data, and work with our Feature Engineering and Rule Development algos and processes. As a result we are able to augment, amplify and scale our clients' knowledge (not replace it).
When engaging Users, our goal is to help them make superior decisions (one at a time). We achieve this by delivering state-of-the-art accuracy, and by providing a detailed explanation of every single decision. One reason for engaging Users early is to identify ways to visually represent explanations.
Semantic Brain has developed specialized tools and processes to identify and prepare the features and rules, which we then integrate with numerous AutoML, AI/ML Libraries, MLOps and Cloud offerings.
What makes BizML unique?
BizML is designed to provide Finance & Marketing Professionals the ability to build and update better performing AI / ML solutions while reducing their dependency on Data Scientists and IT. BizML is able to deliver this by
Delivering additional and pre-processed data to complement customer data
Increases data quantity and quality while reducing Time-to-Market
Capturing Financial / Marketing knowledge to automatically (or semi-automatically) build and update Models
There is greater emphasis knowledge capture to build highest performance models as opposed to automation
Providing AI explanations and feedback in Financial / Marketing language back to Expert(s) / User(s)
How does BizML work?
All our components and processes are designed to optimize the Human to AI feedback loop. Technically we achieve this using
1) Translation: Business Specialist / User to AI, and AI to Business Specialist / User translation.
2) Encapsulation: Hide complexities of ML and make it more business friendly.
More detailed description of components and processes are provided below.
Business Needs & Objectives
BizML engagements with clients start with identifying business needs and objectives. This includes (but is not limited to) identifying potential features and rules from Business Experts' input, and translating business objectives to accuracy, precision, recall, f1-score, AUC, etc.
Data & Knowledge Graph
Semantic Brain acquires and persists data from Financial Markets, Security Filings (e.g. Fundamental Data from Edgar), Government Sources (e.g. Economic Information), News, Social Media (e.g. Company related posts on Twitter), General Info (e.g. Wikipedia). Some of this information is used to derive Technical Indicators, and Financial Ratios. All this information is linked at a high-level in the form of a Semantic Graph Database.
We are continuing to grow our Knowledge Graph, and it is increasingly helping us deliver more value faster to our customers.
BizML solution also works with customers own data.
Feature Identification
BizML uses a structured approach to identifying features, and this includes early formation and validation of hypotheses.
For example, we identified Technical, Fundamental, Economic, Social and Derivative Trading data as influencing stock prices. We formed Hypothesis at Category and Individual Feature / Attribute level; and validated these hypothesis However, our initial models only used Technical and some Fundamental and Economic Data.
Feature Engineering
One of the hypothesis we formed early was that earning surprises could influence stock prices. We identified earnings estimate, actual, difference, surprise % and stock price as being attributes relating to this hypothesis, and engineered many features. These features were then validated using our proprietary algos - resulting in higher accuracy, and reduced compute cycles and time-to-market.
Note: Financial Engineering methodologies were used to engineer Features for stock price prediction.
Rule Discovery
Rule Discovery is an excellent way to optimize Human to AI feedback loop. In the investing / trading space SMEs (Subject Matter Experts) identified Moving Averages, RSI (Relative Strength Indicator) and ATR (Average True Range) as critical to trading decision. One of the expert rules was RSI value of above 70 indicates overbought (i.e. prices likely to go down) and below 30 indicates oversold (i.e. prices likely to go up).
Rule Optimization / Generation
Asperios predicts price movement for 2, 3, 5, 10 and 20 days using EOD (End of Day) data. Rather than manually optimizing and validating rules, BizML uses proprietary Algos to generate and validate rules, and for stock price prediction we assessed RSI Rules for multiple periods of 4, 5, 9, 10, 14, 15, 20 and 21 and selected the best ones.
Feature / Rule Selection
Feature / Rule Selection can be broken down into Individual Feature / Rule Selection and Feature / Rule Set Selection.
Individual Feature / Rule Selection
This proprietary functionality works with Feature Identification, Feature Engineering, Rule Discovery and Rule Optimization / Generation create best and smallest possible superset of features.
Feature / Rule Set Selection
We use SOTA (State Of The Art) 3rd party and open source software to derive one or more subsets that will deliver the most accurate predictions (or precision, accuracy, etc.)
Overfitting Detection
Overfitting is an important problem in ML. BizML uses proprietary approaches to detect and avoid / minimize overfitting - especially for time series forecasting.
Other Capabilities
BizML integrates it proprietary algos and processes with many other state-of-the-art technologies to deliver - Model Selection & Architecture, Hyper-parameter Tuning, Explainability & Visualization and Ongoing Monitoring & Updates.
Technology Stack
BizML was developed using and/or integrates with the best tools and frameworks available on the market today. This is includes (but is not limited to)
AI / ML | Tensorflow, PyTorch, Scikit Learn, Keras, Spark, SpaCy, Hugging Face Transformers, CatBoost, XGBoost, LightGBM, Optuna, MLFlow, Cloud AutoML, TabNet, Turi Create, Vecstack |
Data, Search, Storage | Neo4j, ElasticSearch, Solr, BigQuery, MySQL, Cloud Storage |
Visualization, Front-End | React, Redux, Electron, C3 / D3, Plotly, Streamlit, Dash |
Programming Languages | Python, JavaScript, Java |
IDE | Jupyter, VS Code |
Why did Semantic Brain develop BizML?
Semantic Brain as a part of its core product offering (i.e. Asperios) developed Predictive Analytics capabilities. Asperios as a product delivers solutions to the Investment Research & Analysis market. Forecasting future stock prices is a very challenging problem to solve, especially in dynamic markets such as North America. However, Semantic Brain has been very successful in this endeavour.
We were able to demonstrate 100% Accuracy on Future Top Movers functionality during our first month in production. Future Top Movers is designed to identify stocks that are expected to have significant price changes (up or down), and at a minimum change price by more than 5% within 5 days. While 33% of the stocks within that month moved more than 5%. 100% of the stocks we identified as Future Top Movers moved more than 5% in the first month, and the vast majority moved more than 10%.
Similarly we were able to predict Stock Price Change Direction, and were able to continually improve prediction accuracy to achieve state of the art results.
We decided spin-off BizML as a separate product, upon realizing the value our Predictive Analytics software and processes can deliver in other areas of Finance and Marketing.