Geek Answers Handbook

Django-Haystack using Amazon Elasticsearch with IAM credentials

I hope to use the Amazon Elasticsearch server to provide a search for longtext fields in a Django database. However, I also don’t want to disclose this search to those who don’t have a login, and you don’t want to rely on security through obscurity or some kind of IP restriction tactics (if this does not work with the existing heroku application where the Django application is deployed )

Haystack seems to be important for this, but there seems to be no easy way to configure it to use Amazon IAM credentials to access the Elasticsearch service. This functionality exists in elasticsearch-py, which it uses.

https://elasticsearch-py.readthedocs.org/en/master/#running-with-aws-elasticsearch-service

from elasticsearch import Elasticsearch, RequestsHttpConnection from requests_aws4auth import AWS4Auth host = 'YOURHOST.us-east-1.es.amazonaws.com' awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es') es = Elasticsearch( hosts=[{'host': host, 'port': 443}], http_auth=awsauth, use_ssl=True, verify_certs=True, connection_class=RequestsHttpConnection ) print(es.info())

Regarding the use of HTTP authorization, I found this in the https://github.com/django-haystack/django-haystack/issues/1046 section

 from urlparse import urlparse parsed = urlparse('https://user: pass@host :port') HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine', 'URL': parsed.hostname, 'INDEX_NAME': 'haystack', 'KWARGS': { 'port': parsed.port, 'http_auth': (parsed.username, parsed.password), 'use_ssl': True, } } }

I am wondering if there is a way to combine these two, something like the following (which, as expected, gives an error, since it is more than just a username and password):

 from requests_aws4auth import AWS4Auth awsauth = AWS4Auth([AACCESS_KEY],[SECRET_KEY],[REGION],'es') HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine', 'URL': [AWSHOST], 'INDEX_NAME': 'haystack', 'KWARGS': { 'port': 443, 'http_auth': awsauth, 'use_ssl': True, 'verify_certs': True } }, }

The error is here:

 TypeError at /admin/ must be convertible to a buffer, not AWS4Auth Request Method: GET Request URL: http://127.0.0.1:8000/admin/ Django Version: 1.7.7 Exception Type: TypeError Exception Value: must be convertible to a buffer, not AWS4Auth Exception Location: /usr/lib/python2.7/base64.py in b64encode, line 53

Any ideas on how to do this?

+6
source share
2 answers

You are one step away from success, add connection_class to KWARGS , and everything should work as expected.

 import elasticsearch HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine', 'URL': [AWSHOST], 'INDEX_NAME': 'haystack', 'KWARGS': { 'port': 443, 'http_auth': awsauth, 'use_ssl': True, 'verify_certs': True, 'connection_class': elasticsearch.RequestsHttpConnection, } }, }
+8
source

AWS Identity and Access Management (IAM) allows you to manage users and user permissions for AWS so that users of AWS AWS resources can gain control over them.

You cannot use IAM credentials to authorize users at the application level through http_auth, since it appears that you are trying to do this through Haystack. These are different authentication schemes for different services. They are incompatible.

In your case of using security, you stated the need to: 1) restrict access to your application and 2) protect the Elasticsearch service port from public access. These two requirements can be met using the following methods:

Restrict access to your application

I also do not want to show this search to those who do not have a magazine

For the front-end search application, you want to use the Basic Access Authentication (HTTP auth) server configuration on the web server. Here you want to control user access to your application using the standard username and password http_auth (again, not IAM). This will provide your application at the application level.

Dock Elasticsearch Service Port

They don’t want to rely on security through obscurity or some tactics of limiting IP (if it does not work well with the existing heroku application where the Django application is deployed).

The IP restriction is exactly what will work here and is compatible with AWS best security practices. You want to use security groups and security group rules as a firewall to control traffic for your EC2 instances.

Given the configuration of Haystack:

 HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine', 'URL': 'http://127.0.0.1:9200/', 'INDEX_NAME': 'haystack', }, }

You will want to implement IP address restriction in the security group and / or ACL level on this IP address and on port 127.0.0.1 to restrict access only from your host or other Django authorization hosts. This will provide protection against any unauthorized access at the service level.

In your implementation, the URL will most likely be allowed a public or private IP address depending on your network architecture.

0
source

More articles:


All Articles