Blog

As more lending companies got to use Knowru mainly for their financial risk management, I got to work with Chief Risk Officers in different countries and better understand the differences in their risk management methodologies. In this blog post, I’d like to highlight some environmental differences across countries and these differences’ implications on risk management. In the concluding part, I…

Thanks to its interface getting easier, Deep Learning has become prevalent when analyzing irregular and complex data formats such as text, audio, images and videos. However, when it comes to deploying Deep Learning models, machine learning (ML) engineers found themselves left with not many choices because Deep Learning models require a high hardware specification such as GPU due to their…

1. Code coverage featureLast time, we released a feature to see statistics related to test results (like each field’s statistics (e.g. min, median, max) and number of passes and fails). This time we are excited to release a code coverage feature where you can see what lines of code are tested during your testing on Knowru.In a test scenario result detail page, click the “View This File” button for…

1 . Test results at field levelFrom now, after you run a test, you can see which field has differences.TableAlso, by clicking a field, you can see the distribution of the field without having to do any kind of analysis.Graph2. Fixed bugsWe fixed issues when loading a list of runs from multiple time zonesWe changed the default value of of runnables from to so that versions with 0 weights…

1. Bookmark feature for runnablesYou can bookmark your favorite runnables in order to more easily access them laterBookmark runnableFilter marked runnable2. Supporting new language versionsWe now support R 3.5 and R 3.5 with LightGBM installed. You can enjoy more stable and advanced version of R. Also, we automated installing and configuring LightGBM, a powerful GBM package developed by Microsoft…

1. Download recent result featureYou can easily download recent n number of runs in the run list page.Download recent results2. Support DockerfileWhen creating a runnable, often you need to install system-wide packages. From now on, you can provide your own Dockerfile to install system-wide packages before Python or R packages. For example, installing , the Python package to interact with a…

1. Monitoring featureRegularly check that connections to all runnables are stableIf connections are too slow or result in errors, automatically send emails to managers to start checkingMontioring page2. Fixed the issue when downloading a list of runs containing non-ascii charactersIf any question, please feel free to contact to support@knowru.com.Regards,Knowru Team

1 . Released new frontendSeparated frontend for faster response and better user experience.New Frontend Overview2. Restored the "Download CSV Button"The new frontend was missing the “Download CSV Button” in the Run List pages and we restored it.3. Added “Numeric Error Threshold” in the Test featureTo avoid not passing tests due to very small numeric differences, added the “Numeric error threshold…

You may sometime feel pity that you are creating fabulous models on Python 3, however, not satisfied after putting them into production. And, you probably feeling that you are losing long time on finding and fixing errors after running your models. In that regards, we are excited to share some of improvements we have made in our platform.1. Supporting Python 3.5 and 3.6From now on, Knowru’s…

H2O is regarded as one of the best machine learning platforms out there with its many advanced algorithms, interface to R, Python and Scala, ability to work with BigData frameworks and scalable in-memory architecture. In this blog post, we will see how to create a microservice that makes predictions for given input using a H2O Random forest model.First, let us create a H2O model. Creating a model…

To measure is to know Lord Kevin (1824 – 1903)As we’ve been talking to many companies, we’ve found that one of common reasons that companies do not use machine learning is their fear that a machine learning model would slow their real-time transaction processing. Moreover, some companies that currently use machine learning models in their product environments wanted to know whether changing…

It is important to configure the size of memory to prevent unexpected problems in running your models and work efficiently in finding your errors history. Check that out how you can do it with Knowru.1. Users can configure memory size when creating runnables and runnable versions. Please note that once a runnable's memory size is determined, users won't be able to change to prevent unexpected…

Previously, in order to use external resources that require credentials, you had to write your sensitive information like your password in your scripts. Or, when you wanted to change some configurations inside your APIs, you would have to release a new version, which took a long time.Going forward, you won’t have these issues with our new "Environment Variables" feature.In order not to write…

Dear Fans of Knowru,We are extremely excited to share Knowru’s new features which we believe will greatly enhance your ability to navigate, visualize and manage ML models on Knowru. Please find below for more details.1. Success & Error Run Count graph has now a better look2. Median and 99 percentile response times of all runs are now available on the overview pageBefore, there was no feature to…

As new sets of data come in, it is necessary that you refit and release new versions of your machine learning models. In this article, we share the best practices and tips that will help you go through the release process smoothly.1. Release Gradually (A/B Testing)When you have an existing version, ensure a new version processes only a small portion of transactions in the beginning. Even though…

Dear fans of Knowru,We are very excited to share some of improvements we have made in our today's release.1. Support for CORS (Cross-Origin Resource Sharing)Before making HTTP requests, many frontend frameworks such as vue.js make OPTION requests (i.e. pre-flight requests) to API endpoints to gain permissions (or to see if they indeed have permissions) for their subsequent real requests. This…

Knowing usage is an integral part of managing your APIs. On top of that, Knowru provides various analytics features that you'd love in managing your APIs, such as visualizing distributions of key variables on the fly, setting alarms and monitoring response time. This post will talk about how to use various analytics features that Knowru offers. If you are not familiar with how to create APIs…

Combating online fraud using data science poses unexpected challenges. Find our CEO Ken Park's presentation on "BigData to rescue anomaly detection" in the 4th Predictive Analytics and APIs conference in Boston to find how to address the challenges using machine learning and BigData.BigData to rescue anomaly detection PDF

So far we have written and tested a simple word-count PySpark application in a local environment. Now let us run it across multiple servers using AWS EMR.Step 1. Modify the word-count application for EMROur existing word-count application will run totally fine in an EMR cluster but it simply outputs its result to the standard output, which will make it harder for us to find the result (when…

Congratulations! You have successfully installed PySpark and wrote PySpark applications in your local environment. One of big advantages of utilizing a local environment to develop a Spark application is that it does not involve spawning new servers and distributing data to multiple computers which consequently saves time and money. Nonetheless, testing PySpark applications in a local environment…

Knowing how to write and run Spark applications in a local environment is both essential and crucial because it allows us to develop and test your applications in a cost-effective way. This tutorial will show how to achieve the goal with a simple PySpark script in an Ubuntu environment as an example.Step 0. Install JavaCheck Java is installed in your system.If it says , then Java is not installed…

Apache Spark provides hundreds of configurable parameters to give a room for developers to optimize their applications for specific needs. In developing our anomaly detection application using Spark, we have found that there are 2 configurations that, if set properly, can always improve applications’ performance. Here we share them so that future Spark developers can enhance their applications…

Terms of ServicePrivacy Statement Acceptable Use PolicyIntroductionYour use of the Service is subject to this Acceptable Use Policy. If you are found to be in violation of our policies at any time, as determined by Knowru in its sole discretion, we may warn you or suspend or terminate your account. Please note that we may change our Acceptable Use Policy at any time, and pursuant to the Knowru…

Terms of ServicePrivacy Statement Acceptable Use PolicyIntroductionThis Privacy Statement is effective May 15, 2016.Knowru, LLC. ("Knowru" or the "Company") is committed to protecting the privacy of individuals who visit the Company's Web sites ("Visitors") and individuals who register to use the Services as defined below ("Customers"). This Privacy Statement describes Knowru's Web site privacy…

Terms of ServicePrivacy Statement Acceptable Use Policy1. Your agreement with Knowrua. Your use of the Services is governed by this agreement ( the "Terms"). "Knowru" means Knowru Limited at Unit 1411 Cosco Tower, 183 Queen's Road Central, Sheung Wan, Hong Kong 999077, and its subsidiaries and affiliates involved in providing the Services. The “Services” (or “Services”) means the services Knowru…

Congratulations! Now, you have a spark application that runs in your local environment. Now it is time to scale up and use a multitude of powerful servers to digest really large datasets with the fantastic app you just created. You look around and realize there are multiple options to host your spark application. In our case, we chose EMR, a BigData solution provided by AWS, over others for…

The overall response rate for a marketing campaign by a Portuguese bank was 11.27%. In contrast, the 198 campaigns (among 40K+) to a person who purchased a product the last time on a date when the Euribor rate (the Euro Interbank Offered Rate) is less than 0.7 had 80.30% response rate. This is not the only group; the 1,270 campaigns with contacts via cellular phones to customers who purchased…

How happy would you be knowing ALL segments whose default rate is more than twice of the average? The advent of BigData and cloud technologies has made such credit anomaly (or fraud) detection possible. For instance, our algorithm has found all segmentswith at least 40 customers and 50% default rate from the German credit data. This post suggests a new way of identifying patterns of highly bad…

The last 3 posts explained how to create a credit model, build an API for the model using plumber and scale it up using AWS and Docker:Post 1. How to create a RESTful API for a machine learning credit model in RPost 2. How to scale up machine learning credit model RESTful APIs using DockerPost 3. How to scale up credit model APIs using AWS.These posts demonstrate machine learning models can be…

A bottom-up perspective from a former IT consultantOnly a few years ago, cloud did not have another meaning besides "a visible mass of condensed water vapor floating in the atmosphere, typically high above the ground"Nowadays, when you hear the word "cloud", it leads to a whole new world. Cloud computing and storage have become an integral part of an IT architecture. It is highly scalable and you…

IntroductionUbiquity in e-commerce and online financial services has brought widespread online frauds. Fraudsters typically exploit the fact that merchants or finance service providers cannot see their customers face-to-face and illegally use stolen credit card information or IDs on their behalf.Machine learning has become a very popular tool to fight against these online frauds. So much data has…

Our last post talked about scaling up APIs for R credit models using Docker. What's special about the example API was that it was running using Plumber, a native R package. That was a huge win because the R model did not have to be translated to some other language. Another big advantage was scalability - with Docker, the API could handle concurrent requests much faster (improved average…

limitations we discussed was scalability (i.e. handling lots of requests). This could be particularly difficult for R machine learning models because R is a single-thread language.We dedicate two posts on solving this scalability issue for R web APIs. This one presents ways to scale up your RESTful APIs using Docker, a recent technology that has become popular since its birth, and the next one…

In our last post, we talked about how APIs can solve issues when programmers use different programming languages. For example, R is the language many data scientists use to create machine learning models but is rarely used by other types of programmers. As a remedy for the problem, we will create a RESTful API using only open source libraries and see that other programming languages can interact…

R is the language of choice for many statisticians and data scientists when building predictive models because it has so many packages devoted for predictive modeling. Nevertheless, because most packages in R are for pure data analysis and machine learning purposes, software developers rarely used the language when developing web, mobile, or really any other kind of applications.If your company…