As more lending companies got to use Knowru mainly for their financial risk management, I got to work with Chief Risk Officers in different countries and better understand the differences in their risk management methodologies. In this blog post, I’d like to highlight some environmental differences across countries and these differences’ implications on risk management. In the concluding part, I…
January 15, 2019Thanks to its interface getting easier, Deep Learning has become prevalent when analyzing irregular and complex data formats such as text, audio, images and videos. However, when it comes to deploying Deep Learning models, machine learning (ML) engineers found themselves left with not many choices because Deep Learning models require a high hardware specification such as GPU due to their…
December 04, 20181. Code coverage featureLast time, we released a feature to see statistics related to test results (like each field’s statistics (e.g. min, median, max) and number of passes and fails). This time we are excited to release a code coverage feature where you can see what lines of code are tested during your testing on Knowru.In a test scenario result detail page, click the “View This File” button for…
November 26, 20181 . Test results at field levelFrom now, after you run a test, you can see which field has differences.TableAlso, by clicking a field, you can see the distribution of the field without having to do any kind of analysis.Graph2. Fixed bugsWe fixed issues when loading a list of runs from multiple time zonesWe changed the default value of of runnables from to so that versions with 0 weights…
November 11, 20181. Bookmark feature for runnablesYou can bookmark your favorite runnables in order to more easily access them laterBookmark runnableFilter marked runnable2. Supporting new language versionsWe now support R 3.5 and R 3.5 with LightGBM installed. You can enjoy more stable and advanced version of R. Also, we automated installing and configuring LightGBM, a powerful GBM package developed by Microsoft…
October 29, 20181. Download recent result featureYou can easily download recent n number of runs in the run list page.Download recent results2. Support DockerfileWhen creating a runnable, often you need to install system-wide packages. From now on, you can provide your own Dockerfile to install system-wide packages before Python or R packages. For example, installing , the Python package to interact with a…
October 17, 20181. Monitoring featureRegularly check that connections to all runnables are stableIf connections are too slow or result in errors, automatically send emails to managers to start checkingMontioring page2. Fixed the issue when downloading a list of runs containing non-ascii charactersIf any question, please feel free to contact to support@knowru.com.Regards,Knowru Team
September 17, 20181 . Released new frontendSeparated frontend for faster response and better user experience.New Frontend Overview2. Restored the "Download CSV Button"The new frontend was missing the “Download CSV Button” in the Run List pages and we restored it.3. Added “Numeric Error Threshold” in the Test featureTo avoid not passing tests due to very small numeric differences, added the “Numeric error threshold…
September 02, 2018You may sometime feel pity that you are creating fabulous models on Python 3, however, not satisfied after putting them into production. And, you probably feeling that you are losing long time on finding and fixing errors after running your models. In that regards, we are excited to share some of improvements we have made in our platform.1. Supporting Python 3.5 and 3.6From now on, Knowru’s…
June 14, 2018H2O is regarded as one of the best machine learning platforms out there with its many advanced algorithms, interface to R, Python and Scala, ability to work with BigData frameworks and scalable in-memory architecture. In this blog post, we will see how to create a microservice that makes predictions for given input using a H2O Random forest model.First, let us create a H2O model. Creating a model…
June 01, 2018To measure is to know Lord Kevin (1824 – 1903)As we’ve been talking to many companies, we’ve found that one of common reasons that companies do not use machine learning is their fear that a machine learning model would slow their real-time transaction processing. Moreover, some companies that currently use machine learning models in their product environments wanted to know whether changing…
June 01, 2018It is important to configure the size of memory to prevent unexpected problems in running your models and work efficiently in finding your errors history. Check that out how you can do it with Knowru.1. Users can configure memory size when creating runnables and runnable versions. Please note that once a runnable's memory size is determined, users won't be able to change to prevent unexpected…
May 21, 2018Previously, in order to use external resources that require credentials, you had to write your sensitive information like your password in your scripts. Or, when you wanted to change some configurations inside your APIs, you would have to release a new version, which took a long time.Going forward, you won’t have these issues with our new "Environment Variables" feature.In order not to write…
May 08, 2018Dear Fans of Knowru,We are extremely excited to share Knowru’s new features which we believe will greatly enhance your ability to navigate, visualize and manage ML models on Knowru. Please find below for more details.1. Success & Error Run Count graph has now a better look2. Median and 99 percentile response times of all runs are now available on the overview pageBefore, there was no feature to…
April 10, 2018As new sets of data come in, it is necessary that you refit and release new versions of your machine learning models. In this article, we share the best practices and tips that will help you go through the release process smoothly.1. Release Gradually (A/B Testing)When you have an existing version, ensure a new version processes only a small portion of transactions in the beginning. Even though…
March 14, 2018Dear fans of Knowru,We are very excited to share some of improvements we have made in our today's release.1. Support for CORS (Cross-Origin Resource Sharing)Before making HTTP requests, many frontend frameworks such as vue.js make OPTION requests (i.e. pre-flight requests) to API endpoints to gain permissions (or to see if they indeed have permissions) for their subsequent real requests. This…
February 19, 2018Knowing usage is an integral part of managing your APIs. On top of that, Knowru provides various analytics features that you'd love in managing your APIs, such as visualizing distributions of key variables on the fly, setting alarms and monitoring response time. This post will talk about how to use various analytics features that Knowru offers. If you are not familiar with how to create APIs…
January 07, 2018Combating online fraud using data science poses unexpected challenges. Find our CEO Ken Park's presentation on "BigData to rescue anomaly detection" in the 4th Predictive Analytics and APIs conference in Boston to find how to address the challenges using machine learning and BigData.BigData to rescue anomaly detection PDF
October 25, 2017So far we have written and tested a simple word-count PySpark application in a local environment. Now let us run it across multiple servers using AWS EMR.Step 1. Modify the word-count application for EMROur existing word-count application will run totally fine in an EMR cluster but it simply outputs its result to the standard output, which will make it harder for us to find the result (when…
September 18, 2017Congratulations! You have successfully installed PySpark and wrote PySpark applications in your local environment. One of big advantages of utilizing a local environment to develop a Spark application is that it does not involve spawning new servers and distributing data to multiple computers which consequently saves time and money. Nonetheless, testing PySpark applications in a local environment…
September 17, 2017Knowing how to write and run Spark applications in a local environment is both essential and crucial because it allows us to develop and test your applications in a cost-effective way. This tutorial will show how to achieve the goal with a simple PySpark script in an Ubuntu environment as an example.Step 0. Install JavaCheck Java is installed in your system.If it says , then Java is not installed…
September 16, 2017Apache Spark provides hundreds of configurable parameters to give a room for developers to optimize their applications for specific needs. In developing our anomaly detection application using Spark, we have found that there are 2 configurations that, if set properly, can always improve applications’ performance. Here we share them so that future Spark developers can enhance their applications…
September 09, 2017Terms of ServicePrivacy Statement Acceptable Use PolicyIntroductionYour use of the Service is subject to this Acceptable Use Policy. If you are found to be in violation of our policies at any time, as determined by Knowru in its sole discretion, we may warn you or suspend or terminate your account. Please note that we may change our Acceptable Use Policy at any time, and pursuant to the Knowru…
September 09, 2017Terms of ServicePrivacy Statement Acceptable Use PolicyIntroductionThis Privacy Statement is effective May 15, 2016.Knowru, LLC. ("Knowru" or the "Company") is committed to protecting the privacy of individuals who visit the Company's Web sites ("Visitors") and individuals who register to use the Services as defined below ("Customers"). This Privacy Statement describes Knowru's Web site privacy…
September 09, 2017Terms of ServicePrivacy Statement Acceptable Use Policy1. Your agreement with Knowrua. Your use of the Services is governed by this agreement ( the "Terms"). "Knowru" means Knowru Limited at Unit 1411 Cosco Tower, 183 Queen's Road Central, Sheung Wan, Hong Kong 999077, and its subsidiaries and affiliates involved in providing the Services. The “Services” (or “Services”) means the services Knowru…
September 09, 2017Congratulations! Now, you have a spark application that runs in your local environment. Now it is time to scale up and use a multitude of powerful servers to digest really large datasets with the fantastic app you just created. You look around and realize there are multiple options to host your spark application. In our case, we chose EMR, a BigData solution provided by AWS, over others for…
September 06, 2017The overall response rate for a marketing campaign by a Portuguese bank was 11.27%. In contrast, the 198 campaigns (among 40K+) to a person who purchased a product the last time on a date when the Euribor rate (the Euro Interbank Offered Rate) is less than 0.7 had 80.30% response rate. This is not the only group; the 1,270 campaigns with contacts via cellular phones to customers who purchased…
July 20, 2017How happy would you be knowing ALL segments whose default rate is more than twice of the average? The advent of BigData and cloud technologies has made such credit anomaly (or fraud) detection possible. For instance, our algorithm has found all segmentswith at least 40 customers and 50% default rate from the German credit data. This post suggests a new way of identifying patterns of highly bad…
July 13, 2017The last 3 posts explained how to create a credit model, build an API for the model using plumber and scale it up using AWS and Docker:Post 1. How to create a RESTful API for a machine learning credit model in RPost 2. How to scale up machine learning credit model RESTful APIs using DockerPost 3. How to scale up credit model APIs using AWS.These posts demonstrate machine learning models can be…
June 28, 2017A bottom-up perspective from a former IT consultantOnly a few years ago, cloud did not have another meaning besides "a visible mass of condensed water vapor floating in the atmosphere, typically high above the ground"Nowadays, when you hear the word "cloud", it leads to a whole new world. Cloud computing and storage have become an integral part of an IT architecture. It is highly scalable and you…
May 08, 2017IntroductionUbiquity in e-commerce and online financial services has brought widespread online frauds. Fraudsters typically exploit the fact that merchants or finance service providers cannot see their customers face-to-face and illegally use stolen credit card information or IDs on their behalf.Machine learning has become a very popular tool to fight against these online frauds. So much data has…
April 22, 2017Our last post talked about scaling up APIs for R credit models using Docker. What's special about the example API was that it was running using Plumber, a native R package. That was a huge win because the R model did not have to be translated to some other language. Another big advantage was scalability - with Docker, the API could handle concurrent requests much faster (improved average…
March 23, 2017limitations we discussed was scalability (i.e. handling lots of requests). This could be particularly difficult for R machine learning models because R is a single-thread language.We dedicate two posts on solving this scalability issue for R web APIs. This one presents ways to scale up your RESTful APIs using Docker, a recent technology that has become popular since its birth, and the next one…
March 06, 2017In our last post, we talked about how APIs can solve issues when programmers use different programming languages. For example, R is the language many data scientists use to create machine learning models but is rarely used by other types of programmers. As a remedy for the problem, we will create a RESTful API using only open source libraries and see that other programming languages can interact…
March 01, 2017R is the language of choice for many statisticians and data scientists when building predictive models because it has so many packages devoted for predictive modeling. Nevertheless, because most packages in R are for pure data analysis and machine learning purposes, software developers rarely used the language when developing web, mobile, or really any other kind of applications.If your company…
February 25, 2017