[GitHub] zeppelin pull request #2343: [ZEPPELIN-2538] JDBC completer improvements for...

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin pull request #2343: [ZEPPELIN-2538] JDBC completer improvements for...

prabhjyotsingh
GitHub user tinkoff-dwh opened a pull request:

    https://github.com/apache/zeppelin/pull/2343

    [ZEPPELIN-2538] JDBC completer improvements for work with large meta

    ### What is this PR for?
    There are some problems if meta is large (few schemas, each schema contains 500+ tables etc.).
    Problems:
   
    1. loading is very long
    2. each update takes one connection if updates are long, the situation may arise that the entire pool will be busy
    3. no cache
    This PR solves these problems. Added cache and access by full path (schema.table, schema.table.column)
   
   
    ### What type of PR is it?
    Improvement
   
    ### What is the Jira issue?
    https://issues.apache.org/jira/browse/ZEPPELIN-2538
   
    ### Screenshots (if appropriate)
    ![peek 2017-05-15 15-03](https://cloud.githubusercontent.com/assets/25951039/26054252/4cef6980-3985-11e7-9719-e6138eb777f6.gif)
   
   
    ### Questions:
    * Does the licenses files need update? no
    * Is there breaking changes for older versions? no
    * Does this needs documentation? no


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tinkoff-dwh/zeppelin ZEPPELIN-2538

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/2343.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2343
   
----
commit d600fa16dc5e18c1a6ebdc7de0dfc71b2ddff752
Author: Tinkoff DWH <[hidden email]>
Date:   2017-05-12T12:36:12Z

    [ZEPPELIN-2538] rewrite sql completer to work with large data

commit f26ab5dad61c7ea4881b07e58acec53dadb34bed
Author: Tinkoff DWH <[hidden email]>
Date:   2017-05-15T06:33:46Z

    [ZEPPELIN-2538] fix tests

commit f999488b54769e148cf4fa1134d66b0c55392556
Author: Tinkoff DWH <[hidden email]>
Date:   2017-05-15T07:26:23Z

    Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2538

commit a5788743781d68c3159bd55b68b32d9bc7dc324f
Author: Tinkoff DWH <[hidden email]>
Date:   2017-05-15T07:32:20Z

    [ZEPPELIN-2538] protection long download

commit e770d261cd492dda4167f0c0b5c69916fb744109
Author: Tinkoff DWH <[hidden email]>
Date:   2017-05-15T09:22:39Z

    [ZEPPELIN-2538] update description

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
Github user Leemoonsoo commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    \cc @cloverhearts @astroshim


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user cloverhearts commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    I will test on this pr.
    Thank you for greate feature. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user astroshim commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    I think this feature is really cool.
    BTW I got following exception whenever i try to run sql completion.
   
    ```
    ==> logs/zeppelin-interpreter-jdbc-jdbc.log <==
    ERROR [2017-05-25 01:29:57,441] ({pool-1-thread-5} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
    java.lang.NullPointerException
    at org.apache.zeppelin.jdbc.SqlCompleter.completeKeyword(SqlCompleter.java:407)
    at org.apache.zeppelin.jdbc.SqlCompleter.completeName(SqlCompleter.java:469)
    at org.apache.zeppelin.jdbc.SqlCompleter.complete(SqlCompleter.java:105)
    at org.apache.zeppelin.jdbc.JDBCInterpreter.completion(JDBCInterpreter.java:844)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:136)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:573)
    at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1940)
    at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1925)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
   
    ==> logs/zeppelin-.local.log <==
    ERROR [2017-05-25 01:29:57,443] ({qtp517210187-14} NotebookServer.java[onMessage]:383) - Can't handle message
    org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:493)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:136)
    at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:304)
    at org.apache.zeppelin.notebook.Note.completion(Note.java:664)
    at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:1345)
    at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:330)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:372)
    at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:355)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:488)
    ... 21 more
    ```
   
    Could you take a look at this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user tinkoff-dwh commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    @astroshim
    it is strangely. do you package new jar? 407 line SqlConleter.java it is start of comment (multiline)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user astroshim commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    @tinkoff-dwh Sorry it was my problem.
    Tested and working good!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user tinkoff-dwh commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    Ready to review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user astroshim commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    @cloverhearts Did you test this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user cloverhearts commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    @tinkoff-dwh
    Sorry. My confirmation is late.
    I will left comment for results in until 12 hours.
    I'm sorry for wait.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user cloverhearts commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    If there is no more comment on this, I will merge it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin issue #2343: [ZEPPELIN-2538] JDBC completer improvements for work w...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user jongyoul commented on the issue:

    https://github.com/apache/zeppelin/pull/2343
 
    LGTM. Will merge it now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] zeppelin pull request #2343: [ZEPPELIN-2538] JDBC completer improvements for...

prabhjyotsingh
In reply to this post by prabhjyotsingh
Github user asfgit closed the pull request at:

    https://github.com/apache/zeppelin/pull/2343


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---